The embodiments described herein relate to machine learning programs. More specifically, the embodiments relate to execution plan recompilation of machine learning programs.
Machine learning (ML) is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Declarative ML aims at a flexible specification of ML algorithms in a high-level language, independent of data characteristics and automatic generation of efficient execution plans. In order to provide efficiency and scalability, hybrid runtime execution plans are compiled, ranging from in-memory single node execution to distributed Map Reduce (MR) or Spark computation and data access. This allows for the compilation of highly efficient execution plans based on data and cluster characteristics. However, during runtime, the compilation of an execution plan may be determined to be inefficient due to basing execution plans on inferences associated with data size and sparsity of intermediate results, initially unknown or changing characteristics.